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Abstract 



Linear programming (LP) decoding approximates maximum-likelihood (ML) decoding of a linear 
block code by relaxing the equivalent ML integer programming (IP) problem into a more easily solved 



LP problem. The LP problem is defined by a set of box constraints together with a set of linear 
inequalities called "parity inequalities" that are derived from the constraints represented by the rows 
of a parity-check matrix of the code and can be added iteratively and adaptively. In this paper, we 
first derive a new necessary condition and a new sufficient condition for a violated parity inequality 



in 

constraint, or "cut," at a point in the unit hypercube. Then, we propose a new and effective algorithm to 



generate parity inequalities derived from certain additional redundant parity check (RPC) constraints that 
can eliminate pseudocodewords produced by the LP decoder, often significantly improving the decoder 
error-rate performance. The cut-generating algorithm is based upon a specific transformation of an initial 
parity-check matrix of the linear block code. We also design two variations of the proposed decoder to 
make it more efficient when it is combined with the new cut-generating algorithm. Simulation results 
for several low-density parity-check (LDPC) codes demonstrate that the proposed decoding algorithms 
significantly narrow the performance gap between LP decoding and ML decoding. 
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Index Terms 

Low-density parity-check (LDPC) codes, linear codes, linear programming (LP) decoding, iterative 
decoding, maximum-likelihood (ML) decoding, pseudocodewords. 

I. Introduction 

Low-density parity-check (LDPC) codes were first introduced by Gallager in the 1960s JT], 
together with a class of iterative decoding algorithms. Later, in the 1990s, the rediscovery of 
LDPC codes by MacKay and Neal [2|], 13]] launched a period of intensive research on these codes 
and their decoding algorithms. Significant attention was paid to iterative message-passing (MP) 
decoders, particularly belief propagation (BP) []4] as embodied by the sum-product algorithm 

(spa) m. 

Despite the unparalleled success of iterative decoding in practice, it is quite difficult to analyze 
the performance of such iterative MP decoders due to the heuristic nature of their message update 
rules and their local nature. An alternative approach, linear programming (LP) decoding, was 
introduced by Feldman et al. fl6] as an approximation to maximum-likelihood (ML) decoding. 

Many theoretical and empirical observations suggest similarities between the performance of 
LP and MP decoding methods. For example, graph-cover decoding can be used as a theoretical 
tool to show the connection between LP decoding and iterative MP decoding 0. 

However, there are some key differences that distinguish LP decoding from iterative MP 
decoding. One of these differences is that the LP decoder has the ML certificate property, i.e., 
it is detectable if the decoding algorithm fails to find an ML codeword. When it fails to find an 
ML codeword, the LP decoder finds a non-integer solution, commonly called a pseudocodeword. 
Another difference is that while adding redundant parity checks satisfied by all the codewords 
can only improve LP decoding, adding redundant parity checks may have a negative effect on 
MP decoding, especially in the waterfall region, due to the creation of short cycles in the Tanner 
graph. This property of LP decoding allows improvements by tightening the LP relaxation, i.e., 
reducing the feasible space of the LP problem by adding more linear constraints from redundant 
parity checks. 
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In the original formulation of LP decoding proposed by Feldman et ah, the number of 
constraints in the LP problem is linear in the block-length but exponential in the maximum check 
node degree, and the authors also argued that the number of useful constraints could be reduced 
to polynomial in code length. The computational complexity of the original LP formulation 
therefore can be prohibitively high, motivating the design of computationally simplified decoding 
algorithms that can achieve the same error-rate performance with a smaller number of constraints. 
For example, efficient polynomial-time algorithms can be used for solving the original LP 
formulation flS). An alternative LP formulation whose size is linear in the check node degree 
and code length can also be obtained by changing the graphical representation of the code [|9), 
ifTOl : namely, all check nodes of high degree are replaced by dendro- subgraphs (trees) with 
an appropriate number of auxiliary degree-3 check nodes and degree-2 variable nodes. Several 
other low-complexity LP decoders were also introduced in flTTTl . suggesting that LP solvers with 
complexity similar to the min-sum algorithm and the sum-product algorithm are feasible. 

Another approach is to add linear constraints in an adaptive and selective way during the LP 
formulation [12]. Such an adaptive linear programming (ALP) decoding approach also allows the 
adaptive incorporation of linear constraints generated by redundant parity checks (RPC) into the 
LP problem, making it possible to reduce the feasible space and improve the system performance. 
A linear inequality derived from an RPC that eliminates a pseudocodeword solution is referred 
to as a "cut." 

An algorithm proposed in lfT2l uses a random walk on a subset of the code factor graph to 
find these RPC cuts. However, the random nature of this algorithm limits its efficiency. In fact, 
experiments show that the average number of random trials required to find an RPC cut grows 
exponentially with the length of the code. 

Recently, the authors of [13] proposed a separation algorithm that derives Gomory cuts from 
the IP formulation of the decoding problem and finds cuts from RPCs which are generated 
by applying Gaussian elimination to the original parity-check matrix. In [14], a cutting-plane 
method was proposed to improve the fractional distance of a given binary parity-check matrix 
- the minimum weight of nonzero vertices of the fundamental polytope - by adding redundant 
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rows obtained by converting the parity-check matrix into row echelon form after a certain 
column permutation. However, we have observed that the RPCs obtained by the approach 
in 031 are not able to produce enough cuts to improve the error-rate performance relative 
to the separation algorithm when they are used in conjunction with either ALP decoding or the 
separation algorithm. A detailed survey on mathematical programming approaches for decoding 
binary linear codes can be found in [fT31 . 

In this paper, we greatly improve the error-correcting performance of LP decoding by designing 
algorithms that can efficiently generate cut-inducing RPCs and find possible cuts from such RPCs. 
First, we derive a new necessary condition and a new sufficient condition for a parity-check 
to provide a cut at a given pseudocodeword. These conditions naturally suggest an efficient 
algorithm that can be used to find, for a given pseudocodeword solution to an LP problem, 
the unique cut (if it exists) among the parity inequalities associated with a parity check. This 
algorithm was previously introduced by Taghavi et al. |[T6l Algorithm 2] and, independently and 
in a slightly different form, by Wadayama [17, Fig. 6]. 

The conditions also serve as the motivation for a new, more efficient adaptive cut-inducing 
RPC generation algorithm that identifies useful RPCs by performing specific elementary row 
operations on the original parity-check matrix of the binary linear code. By adding the cor- 
responding linear constraints into the LP problem, we can significantly improve the error-rate 
performance of the LP decoder, even approaching the ML decoder performance in the high- 
SNR region for some codes. Finally, we modify the ALP decoder to make it more efficient 
when being combined with the new cut-generating algorithm. Simulation results demonstrate 
that the proposed decoding algorithms significantly improve the error-rate performance of the 
original LP decoder. 

The remainder of the paper is organized as follows. In Section HH we review the original 
formulation of LP decoding and several adaptive LP decoding algorithms. Section [III] presents 
the new necessary condition and new sufficient condition for a parity-check to induce a cut, as 
well as their connection to the efficient cut-search algorithm. In Section [IV] we describe our 
proposed algorithm for finding RPC -based cuts. Section [V] presents our simulation results, and 
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II. Linear Programming Decoding and Adaptive Variants 
A. Linear Programming (LP) Relaxation of Maximum Likelihood (ML) Decoding 

Consider a binary linear block code C of length n and a corresponding m x n parity- 
check matrix H. A codeword y G C is transmitted across a memoryless binary-input output- 
symmetric channel, resulting in a received vector r. Assuming that the transmitted codewords 
are equiprobable, the ML decoder finds the solution to the following optimization problem (see, 
e.g., E2) 

minimize *v T u 

(1) 

subject to u G C 

where m G {0, 1}, and 7 is the vector of log-likelihood ratios (LLR) defined as 

Pr{R i = r i \u i = 0) \ 
Pr(i^ = r;K = l); ' 

Since the ML decoding problem (OQ) is an integer programming problem, it is desirable to 
replace its integrality constraints with a set of linear constraints, transforming the IP problem 
into a more readily solved LP problem. The desired feasible space of the corresponding LP 
problem should be the codeword polytope, i.e., the convex hull of all the codewords in C. With 
this, unless the cost vector of the LP decoding problem is orthogonal to a face of the constraint 
polytope, the optimal solution is one integral vertex of its codeword polytope, in which case 
it is the same as the output of the ML decoder. When the LP solution is not unique, there is 
at least one integral vertex corresponding to an ML codeword. However, the number of linear 
constraints typically needed to represent the codeword polytope increases exponentially with the 
code length, which makes such a relaxation impractical. 

As an approximation to ML decoding, Feldman et al. H, (8]| relaxed the codeword polytope 
to a polytope now known as fundamental polytope 0, denoted as V(H), which depends on the 
parity-check matrix H. 



H = log 
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Definition 1 (Fundamental polytope 17]): Let us define 

Cj = {xG F"|(x,h,> =0 (in F 2 )} 
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(3) 



where hj is the jth row of the parity-check matrix H and 1 < j < m. Thus, Cj is the set of 
all binary vectors satisfying the jth parity-check constraint. We denote by conv(Cj) the convex 
hull of Cj in W 1 , which consists of all possible real convex combinations of the points in Cj, 
now regarded as points in W n . The fundamental polytope V(H) of the parity-check matrix H is 
defined to be the set 

m 

7>(H) = P|conv(C,). (4) 

3=1 

Therefore, LP decoding can be written as the following optimization problem: 

minimize ~/ T u 

(5) 

subject to u G P(H). 

The solution of the above LP problem corresponds to a vertex of the fundamental polytope that 
minimizes the cost function. Since the fundamental polytope has both integral and nonintegral 
vertices, with the integral vertices corresponding exactly to the codewords of C [61, 0, if the 
LP solver outputs an integral solution, it must be a valid codeword and is guaranteed to be an 
ML solution, which is called the ML certificate property. The nonintegral solutions are called 
pseudocodewords. Since the fundamental polytope is a function of the parity-check matrix H 
used to represent the code C, different parity-check matrices for C may have different fundamental 
polytopes. Therefore, a given code has many possible LP-based relaxations, and some may be 
better than others when used for LP decoding. 

The fundamental polytope can also be described by a set of linear inequalities, obtained as 
follows O. First of all, for a point u within the fundamental polytope, it should satisfy the box 
constraints such that < u% < 1, for i = 1, . . . , n. Then, let Q {1,2, ... ,n} be the set of 
neighboring variable nodes of the check node j in the Tanner graph, that is, = {i : Hj ti = 1} 
where Hj i is the element in the jth row and iih. column of the parity-check matrix, H. For each 
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row j = 1, . . . ,m of the parity-check matrix, corresponding to a check node in the associated 
Tanner graph, the linear inequalities used to form the fundamental polytope "P(H) are given by 

J2 (! - « + Yl Ui - for a11 V - with l V l odd ( 6 ) 

ieV i€A/"(.7')\V 

where for a set denotes its cardinality. It is easy to see that © is equivalent to 

J^u, - u ^ \ V \ ~ for a11 V ^ JV0')» with l V l odd - ( 7 ) 

iev ieA^(i)\v 

Note that, for each check node j, the corresponding inequalities in © or © and the linear box 
constraints exactly describe the convex hull of the set Cj. 

The linear constraints in © (and therefore also ©) are referred to as parity inequalities, 
which are also known as forbidden set inequalities [6J. It can be easily verified that these linear 
constraints are equivalent to the original parity-check constraints when each Uj takes on binary 
values only. 

Proposition 1 (Theorem 4 in j^jj): The parity inequalities of the form © derived from all 
rows of the parity-check matrix H and the box constraints completely describe the fundamental 
polytope P(H). 

With this, LP decoding can also be formulated as follows 

minimize 7 T u 

subject to < Ui < 1, for all i; 

(8) 

£ Ul >l 

■iev i&hf{j)\V 

for all j,V C Af(j), with |V| odd. 

In the following parts of this paper, we refer to the above formulation of LP decoding 
problem based on the fundamental polytope of the original parity-check matrix as the original 
LP decoding. 
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B. Adaptive Linear Programming (ALP) Decoding 

In the original formulation of LP decoding presented in [0, every check node j generates 
2l^0')l-i parity inequalities that are used as linear constraints in the LP problem described in 
d8]). The total number of constraints and the complexity of the original LP decoding problem 
grows exponentially with the maximum check node degree. So, even for binary linear codes 
with moderate check degrees, the number of constraints in the original LP decoding could be 
prohibitively large. In the literature, several approaches to reducing the complexity of the original 
LP formulation have been described [81- lfT2l . We will use adaptive linear programming (ALP) 
decoding [fT2l as the foundation of the improved LP decoding algorithms presented in later 
sections. The ALP decoder exploits the structure of the LP decoding problem, reflected in the 
statement of the following lemma. 

Lemma 1 (Theorem 1 in [12]): If at any given point u G [0, l] n , one of the parity inequalities 
introduced by a check node j is violated, the rest of the parity inequalities from this check node 
are satisfied with strict inequality. 

Definition 2: Given a parity-check node j, a set V C N(j) of odd cardinality, and a vector 
u G [0, l] n such that the corresponding parity inequality of the form © or © does not hold, 
we say that the constraint is violated or, more succinctly, a cut at u. Q 

In lfT2l . an efficient algorithm for finding cuts at a vector u G [0, l] n was presented. It relies 
on the observation that violation of a parity inequality © at u implies that 

|V|-l<^«i<|V| (9) 

iev 

and 

< u * <u v> for a11 v eV. (10) 

where V is an odd-sized subset of M(j). 

'in the terminology of 1 15], if does not hold for a pseudocodeword u, then the vector (r, t) 6 R n x R, where r» = 1 for 
all i £ V, n — — 1 for all i 6 Af(j)\V, r t — otherwise, and t — \V\ — 1, is a valid cut, separating u from the codeword 
polytope. 
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Given a parity check j, the algorithm first puts its neighboring variables in u into non- 
increasing order, i.e., Uj 1 > ■ ■ ■ > Uj n , for Uj. G It then successively considers subsets 
of odd cardinality having the form V = {?%, . . . ,Uj 2k+1 } C Af(j), increasing the size of V by 
two each step, until a cut (if one exists) is found. This algorithm can find a cut among the 
constraints corresponding to a check node j by examining at most \J\f(j)\/2 inequalities, rather 
than exhaustively checking all 2^^~ 1 inequalities in the original formulation of LP decoding. 

The ALP decoding algorithm starts by solving the LP problem with the same objective function 
as CQ), but with only the following constraints 

< Ui if 74 > 

(11) 

Ui < 1 if 7i < 0. 

The solution of this initial LP problem can be obtained simply by making a hard decision on the 
components of a received vector. The ALP decoding algorithm starts with this point, searches 
every check node for cuts, adds all the cuts found during the search as constraints into the LP 
problem, and solves it again. This procedure is repeated until an optimal integer solution is 
generated or no more cuts can be found (see [fl2l for more details). Adaptive LP decoding has 
exactly the same error-correcting performance as the original LP decoding. 

III. Cut Conditions 

In this section, we derive a necessary condition and a sufficient condition for a parity inequality 
to be a cut at u 6 [0, l] n . We also show their connection to the efficient cut-search algorithm 
proposed by Taghavi et al. [fT6l Algorithm 2] and Wadayama [fT71 Fig. 6]. This algorithm is 
more efficient than the search technique from lfl2l that was mentioned in Section lU 

Consider the original parity inequalities in © given by Feldman et al. in [|6]|. If a parity 
inequality derived from check node j induces a cut at u, the cut can be written as 

(1 — ui) + < 1, for some V C Af(j) with |V| odd. (12) 
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From (fl"2l) and Lemma [T] we can derive the following necessary condition for a parity-check 
constraint to induce a cut. 

Theorem 1: Given a nonintegral vector u and a parity check j, let S = {i E M(j)\0 < Ui < 1} 
be the set of nonintegral neighbors of j in the Tanner graph, and let T = {i E J\f(j)\ui > |}. 
A necessary condition for parity check j to induce a cut at u is 



i&T i£jV{j)\T 



(13) 



This is equivalent to 



— Ui 



>- 2 -\s\-i 



(14) 



where, for i6K, \x\ denotes the absolute value. 

Proof: For a given vector u and a subset X C ftf(j), define the function 



(/(#) = £ (i-to+ 5Z m - 

If parity-check j incudes a cut at u, there must be a set V C Af(j) of odd cardinality such that 
(fl"2l) holds. This means that (7 (V cut ) < 1. Now, it is easy to see that the set T minimizes the 
function g (X), from which it follows that g (T) < g (V cu t) < 1- Therefore, inequality (TT3T ) must 
hold in order for parity check j to induce a cut. 

For I < Ui < 1, we have 



1 


1 











-H 


— 1 


2 ~ 


2 




2 / 



it,- 



and for < Ui < \, we have 



1 

2 



Hence, (1131) can be rewritten as 



1 

Ui 

2 



Ui. 





1 




Uj 




2 



< 1 
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or equivalently, 



1 

Uj 

2 



< 1 



which implies inequality (I141) . ■ 
Remark 1: Theorem \T\ shows that to see whether a parity-check node could provide a cut at 

a pseudocodeword u we only need to examine its fractional neighbors. 

Reasoning similar to that used in the proof of Theorem [T] yields a sufficient condition for a 

parity-check node to induce a cut at u. 

Theorem 2: Given a nonintegral vector u and a parity check j, let S = {i E Af(j)\0 < Ui < 1} 

and T = {i E N(j)\ui > |}. If the inequality 



(1 - m) + M * + 2 

i&N{j)\T 



mm 



1 

Ui 

2 



< 1 



(15) 



holds, there must be a violated parity inequality derived from parity check j. This sufficient 
condition can be written as 



1 

Ui 

2 



2 ■ min 

ieS 



1 



2 



>i.„, 



I. 



(16) 



Proof: Lemma CD implies that, if parity check j gives a cut at u, then there is at most one 
odd-sized set V C J\f(j) that satisfies (fl2)) . From the proof of Theorem [Q we have g (T) < g (X) 
for all X C ftf(j). If |T| is even, we need to find one element z* G A/"(j) such that inserting 
it into or removing it from T would result in the minimum increment to the value of g (T). 

Obviously, i* = arg min ~ — Ui I , and the increment is 2- ~ — u^ I . If more than one i minimizes 

ieAf(j) 1 ^ 1 i^i 

the expression |~ — wj, we choose one arbitrarily as i*. Hence, setting 



V 



T\{i*}, if i* eT 
Tu{i*}, ifi*^T 



we have g (V) = g (T) + 2 ■ \± - u^\ > g (T). If inequality £[5]) holds, then g (T) < g (V) < 1. 
Since either |T| or |V| is odd, (fl"5l) is a sufficient condition for parity-check constraint j to 
induce a cut at u. Arguing as in the latter part of the proof of Theorem [T] it can be shown that 
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(fT5T ) is equivalent to (OH). ■ 
Theorem \T\ and Theorem |2] provide a necessary condition and a sufficient condition, respec- 
tively, for a parity-check node to produce a cut at any given vector u. It is worth pointing out that 
(fl"3l) becomes a necessary and sufficient condition for a parity check to produce a cut when |T| 
is odd, and (fl"5l) becomes a necessary and sufficient condition when |T| is even. Together, they 
suggest a highly efficient technique for finding cuts, the Cut-Search Algorithm (CSA) described 
in Algorithm \T\ If there is a violated parity inequality, the CSA returns the set V corresponding 
to the cut; otherwise, it returns an empty set. 

As mentioned above, the CSA was used by Taghavi et al. [16, Algorithm 2] in conjunction 
with ALP decoding, and by Wadayama ifTTl Fig. 6] as a feasibility check in the context of 
interior point decoding. In addition to providing another perspective on the CSA, the necessary 
condition and sufficient condition proved in Theorems 1 and 2, respectively, serve as the basis 
for a new adaptive approach to finding cut-inducing RPCs, as described in the next section. 

IV. LP Decoding with Adaptive Cut- Generating Algorithm 
A. Generating Redundant Parity Checks 

Although the addition of a redundant row to a parity-check matrix does not affect the F 2 - 
nullspace and, therefore, the linear code it defines, different parity-check matrix representations 
of a linear code may give different fundamental polytopes underlying the corresponding LP 
relaxation of the ML decoding problem. This fact inspires the use of cutting-plane techniques 
to improve the error-correcting performance of the original LP and ALP decoders. Specifically, 
when the LP decoder gives a nonintegral solution (i.e., a pseudocodeword), we try to find the 
RPCs that introduce cuts at that point, if such RPCs exist. The cuts obtained in this manner are 
called RPC cuts. The effectiveness of this method depends on how closely the new relaxation 
approximates the ML decoding problem, as well as on the efficiency of the technique used to 
search for the cut-inducing RPCs. 

An RPC can be obtained by modulo-2 addition of some of the rows of the original parity- 
check matrix, and this new check introduces a number of linear constraints that may give a cut. 



SUBMITTED TO IEEE TRANS. INFORM. THEORY 



13 



Algorithm 1 Cut-Search Algorithm (CSA) 
Input: parity-check node j and vector u 

Output: variable node set V 

i: V^T={ie Af(j)\ui > \} and S <- {i e Af(j)\0 < < 1} 
2: if | V| is even then 
3: if S ^ then 

4: %* <r- arg min \ \ — u\ 

i<=S 1 z 1 

5: else 

6: i* <- arbitrary % e N(j) 

7: end if 

8: Hi* eV then 

9: V^V\{i*} 

10: else 

11: V^VU{f} 

12: end if 
13: end if 

14: if J2 (1 - « + E M i < 1 then 

«ev ieA/"(j)\v 
15: Found the violated parity inequality on parity-check node j 

16: else 

17: There is no violated parity inequality on parity-check node j 

18: V <- 

19: end if 
20: return V 
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In lfT2l . a random walk on a cycle within the subgraph defined by the nonintegral entries in a 
pseudocodeword served as the basis for a search for RPC cuts. However, there is no guarantee 
that this method will find a cut (if one exists) within a finite number of iterations. In fact, the 
average number of random trials needed to find an RPC cut grows exponentially with the code 
length. 

The IP-based separation algorithm in [[T3l performs Gaussian elimination on a submatrix 
comprising the columns of the original parity-check matrix that correspond to the nonintegral 
entries in a pseudocodeword in order to get redundant parity checks. In lfT4l . the RPCs that 
potentially provide cutting planes are obtained by transforming a column-permuted version of 
the submatrix into row echelon form. The chosen permutation organizes the columns according 
to descending order of their associated nonintegral pseudocodeword entries, with the exception 
of the column corresponding to the largest nonintegral entry, which is placed in the rightmost 
position of the submatrix |[T4l p. 1010]. This approach was motivated by the fact that a parity 
check j provides a cut at a pseudocodeword if there exists a variable node in Af(j) whose value 
is greater than the sum of the values of all of the other neighboring variable nodes |[T4l Lemma 
2]. However, when combined with ALP decoding, the resulting "cutting -plane algorithm" does 
not provide sufficiently many cuts to surpass the separation algorithm in error-rate performance. 

Motivated by the new derivation of the CSA based on the conditions in Theorems Q] and 
[21 we next propose a new algorithm for generating cut-inducing RPCs. When used with ALP 
decoding, the cuts have been found empirically to achieve near-ML decoding performance in 
the high-SNR region for several short-to-moderate length LDPC codes. However, application of 
these new techniques to codes with larger block lengths proved to be prohibitive computationally, 
indicating that further work is required to develop practical methods for enhanced LP decoding 
of longer codes. 

Given a nonintegral solution of the LP problem, we can see from Theorems \T\ and |2] that 
an RPC with a small number of nonintegral neighboring variable nodes may be more likely 
to satisfy the necessary condition for providing a cut at the pseudocodeword. Moreover, the 
nonintegral neighbors should have values either close to or close to 1; in other words, they 
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should be as far from ~ as possible. 

Let p = (pi,P2, ■ ■ ■ ,p n ) ^ [0,1]™ be a pseudocodeword solution to LP decoding, with a 
nonintegral positions, b zeros, and n — a — b ones. We first group entries of p according to 
whether their values are nonintegral, zero, or one. Then, we sort the nonintegral positions in 
ascending order according to the value of || — pi\ and define the permuted vector p' = n(p) 
satisfying the following ordering 



< ... < 



P\ 



(17) 



a+l 



Pa+b = °> 



and 



Pa+b+l 



Pn 



1. 



By applying the same permutation II to the columns of the original parity-check matrix H, we 
get 

H'4n(H) = (HW|H^|H«) (18) 

where H>% and consist of columns of H corresponding to positions of p' with 

nonintegral values, zeros, and ones, respectively. 

The following familiar definition from matrix theory will be useful [fT8l p. 10]. 

Definition 3: A matrix is in reduced row echelon form if its nonzero rows (i.e., rows with 
at least one nonzero element) are above any all-zero rows, and the leading entry (i.e., the first 
nonzero entry from the left) of a nonzero row is the only nonzero entry in its column and is 
always strictly to the right of the leading entry of the row above it. 

By applying a suitable sequence of elementary row operations $ (over F 2 ) to H', we get 



H^$(H0 = (H( f )|H(°)|H«) 



(19) 



where is in reduced row echelon form. Applying the inverse permutation II 1 to columns 
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of H, we get an equivalent parity-check matrix 

H = n _1 (H) (20) 

whose rows are likely to be cut-inducing RPCs, for the reasons stated above. 

Multiple nonintegral positions in the pseudocodeword p could have values of the same distance 
from ~, i.e., || — pi\ = |~ — pj\ for some i ^ j. In such a case, the ordering of the nonintegral 
positions in (fTTT ) is not uniquely determined. Hence, the set of RPCs generated by operations 
(fl"8T)-(l2"0"l) may depend upon the particular ordering reflected in the permutation II. Nevertheless, 
if the decoder uses a fixed, deterministic sorting rule such as, for example, a stable sorting 
algorithm, then the decoding error probability will be independent of the transmitted codeword. 

The next theorem describes a situation in which a row of H is guaranteed to provide a cut. 

Theorem 3: If there exists a weight-one row in submatrix ftw, the corresponding row of the 
equivalent parity-check matrix H is a cut-inducing RPC. 

Proof: Given a pseudocodeword p, suppose the jth row of submatrix fi( f ) has weight one 
and the corresponding nonintegral position in p is pi. Since it is the only nonintegral position in 
N(j), the left-hand side of (fT6b is equal to — ~ — p 4 |. Since < Pi < 1, this is larger than — |, 
the right-hand side. Hence, according to Theorem |2] RPC j satisfies the sufficient condition for 
providing a cut. In other words, there must be a violated parity inequality induced by RPC j. ■ 

Remark 2: Theorem |3] is equivalent to ffT3l Theorem 3.3]. The proof of the result shown here, 
though, is considerably simpler, thanks to the application of Theorem |2] 

Although Theorem |3] only ensures a cut for rows with weight one in submatrix H®, rows in 
of weight larger than one may also provide RPC cuts. Hence, the CSA should be applied 
on every row of the redundant parity-check matrix H to search for all possible RPC cuts. The 
approach of generating a redundant parity-check matrix H based on a given pseudocodeword 
and applying the CSA on each row of this matrix is called adaptive cut generation (ACG). 
Combining ACG with ALP decoding, we obtain the ACG- ALP decoding algorithm described in 
Algorithm |2] Beginning with the original parity-check matrix, the algorithm iteratively applies 
ALP decoding. When a point is reached when no further cuts can be produced from the original 
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parity-check matrix, the ACG technique is invoked to see whether any RPC cuts can be generated. 
The ACG- ALP decoding iteration stops when no more cuts can be found either from the original 
parity-check matrix or in the form of redundant parity checks. 

Algorithm 2 Adaptive Linear Programming with Adaptive Cut-Generation (ACG-ALP) Decod- 
ing Algorithm 

Input: cost vector 7, original parity-check matrix H 
Output: Optimal solution of current LP problem 

l: Initialize the LP problem with the constraints in (TTTb . 

2: Solve the current LP problem, and get optimal solution x*. 

3: Apply Algorithm 1 (CSA) on each row of H. 

4: if No cut is found and x* is nonintegral then 

5: Construct H associated with x* according to (|T~8T> — (120T> . 

6: Apply Algorithm 1 (CSA) to each row of H. 

7: end if 

8: if No cut is found then 
9: Terminate. 
10: else 

11: Add cuts that are found into the LP problem as constraints, and go to line 2. 
12: end if 



B. Reducing the Number of Constraints in the LP Problem 

In the ALP decoding, the number of constraints in the LP problem grows as the number of 
iterations grows, increasing the complexity of solving the LP problem. For ACG-ALP decoding, 
this problem becomes more severe since the algorithm generates additional RPC cuts and uses 
more iterations to successfully decode inputs on which the ALP decoder has failed. 

From Lemma \T\ we know that a binary parity-check constraint can provide at most one cut. 
Hence, if a binary parity check gives a cut, all other linear inequalities introduced by this parity 
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check in previous iterations can be removed from the LP problem. The implementation of this 
observation leads to a modified ALP (MALP) decoder referred to as the MALP-A decoder [16]. 
This decoder improves the efficiency of ALP decoding, where only cuts associated with the 
original parity-check matrix are used. However, with ACG-ALP decoding, different RPCs may 
be generated adaptively in every iteration and most of them give only one cut throughout the 
sequence of decoding iterations. As a result, when MALP-A decoding is combined with the 
ACG technique, only a small number of constraints are removed from the LP problem, and the 
decoding complexity is only slightly reduced. 

Definition 4: A linear inequality constraint of the form ax > b is called active at point x* if 
it holds with equality, i.e., ax* = b, and is called inactive otherwise. 

For an LP problem with a set of linear inequality constraints, the optimal solution x* G [0, l] n 
is a vertex of the polytope formed by the hyperplanes corresponding to all active constraints. In 
other words, if we set up an LP problem with only those active constraints, the optimal solution 
remains the same. Therefore, a simple and intuitive way to reduce the number of constraints is 
to remove all inactive constraints from the LP problem at the end of each iteration, regardless 
of whether or not the corresponding binary parity check generates a cut. This approach is called 
MALP-B decoding IfTBI . By combining the ACG technique and the MALP-B algorithm, we 
obtain the ACG-MALP-B decoding algorithm. It is similar to the ACG-ALP algorithm described 
in Algorithm |2] but includes one additional step that removes all inactive constraints from the 
LP problem, as indicated in Line 3 of Algorithm [3] 

Since adding further constraints into an LP problem reduces the feasible space, the minimum 
value of the cost function is non-decreasing as a function of the number of iterations. In our 
computer simulations, the ACG-MALP-B decoding algorithm was terminated when no further 
cuts could be found. (See Fig. [6] for statistics on the average number of iterations required to 
decode one codeword of the (155,64) Tanner LDPC code.) 

In our implementation of both MALP-B and ACG-MALP-B decoding, we have noticed that 
a considerable number of the constraints deleted in previous iterations are added back into the 
LP problem in later iterations, and, in fact, many of them are added and deleted several times. 
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Algorithm 3 ACG-MALP-B/C Decoding Algorithm 
Input: cost vector 7, original parity-check matrix H 

Output: Optimal solution of current LP problem 

l: Initialize LP problem with the constraints in (fTTT ). 

2: Solve the current LP problem, get optimal solution x*. 

3: ACG-MALP-B only: remove all inactive constraints from the LP problem. 

4: ACG-MALP-C only: remove inactive constraints that have above-average slack values from 

the LP problem. 

5: Apply CSA only on rows of H that have not introduced constraints. 
6: if No cut is found and x* is nonintegral then 
7: Construct H according to x* 
8: Apply CSA on each row of H. 
9: end if 

10: if No cut is found then 
11: Terminate. 
12: else 

13: Add found cuts into LP problem as constraints, and go to line 2. 
14: end if 



We have observed that MALP-B-based decoding generally takes more iterations to decode a 
codeword than ALP-based decoding, resulting in a tradeoff between the number of iterations 
and the size of the constituent LP problems. MALP-B-based decoding has the largest number 
of iterations and the smallest LP problems to solve in each iteration, while ALP-based decoding 
has a smaller number of iterations but larger LP problems. 

Although it is difficult to know in advance which inactive constraints might become cuts in 
later iterations, there are several ways to find a better tradeoff between the MALP-B and ALP 
techniques to speed up LP decoding. This tradeoff, however, is highly dependent on the LP 
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solver used in the implementation. For example, we used the Simplex solver from the open- 
source GNU Linear Programming Kit (GLPK) |fT9~l , and found that the efficiency of iterative 
ALP-based decoders is closely related to the total number of constraints used to decode one 
codeword, i.e., the sum of the number of constraints used in all iterations. This suggests a new 
criterion for the removal of inactive constraints whose implementation we call the MALP-C 
decoder. 

In MALP-C decoding, instead of removing all inactive constraints from the LP problem in 
each iteration, we remove only the linear inequality constraints with slack variables that have 
above-average values, as indicated in Line 4 of Algorithm [3] The ACG-MALP-B and ACG- 
MALP-C decoding algorithms are both described in Algorithm [3l differing only in the use of 
Line 3 or Line 4. Although all three of the adaptive variations of LP decoding discussed in 
this paper - ALP, MALP-B, and MALP-C - have the exact same error-rate performance as the 
original LP decoder, they may lead to different decoding results for a given received vector when 
combined with the ACG technique, as shown in the next section. 

V. Numerical Results 

To demonstrate the improvement offered by our proposed decoding algorithms, we com- 
pared their error-correcting performance to that of ALP decoding (which, again, has the same 
performance as the original LP decoding), BP decoding (two cases, using the sum-product 
algorithm with a maximum of 100 iterations and 1000 iterations, respectively), the separation 
algorithm (SA) [13J, the random-walk-based RPC search algorithm [[121 . and ML decoding for 
various LDPC codes on the additive white Gaussian noise (AWGN) channel. We use the Simplex 
algorithm from the open-source GLPK |[T9l as our LP solver. The LDPC codes we evaluated 
are MacKay's rate-|, (3,6)-regular LDPC codes with lengths 96 and 408, respectively [|20l : a 
rate-|, (3,4)-regular LDPC code of length 100; the rate-|, (3,5)-regular Tanner code of length 
155 ED; and a rate-0.89, (3,27)-regular high-rate LDPC code of length 999 (201. 

The proposed ACG- ALP, ACG-MALP-B, and ACG-MALP-C decoding algorithms are all 
based on the underlying cut-searching algorithm (Algorithm [Q) and the adaptive cut-generation 
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technique of Section IIV-A1 Therefore, their error-rate performance is very similar. However, 
their performance may not be identical, because cuts are found adaptively from the output 
pseudocodewords in each iteration and the different sets of constraints used in the three proposed 
algorithms may lead to different solutions of the corresponding LP problems. 

In our simulation, the LP solver uses double-precision floating-point arithmetic, and therefore, 
due to this limited numerical resolution, it may round some small nonzero vector coordinate 
values to or output small nonzero values for vector coordinates which should be 0. Similar 
rounding errors may occur for coordinate values close to 1. Coordinates whose values get rounded 
to integers by the LP solver might lead to some "false" cuts - parity inequalities not actually 
violated by the exact LP solution. This is because such rounding by the LP solver would decrease 
the left-hand side of parity inequality ©. On the other hand, when coordinates that should have 
integer values are given nonintegral values, the resulting errors would increase the left-hand side 
of parity inequality ©, causing some cuts to be missed. Moreover, this would also increase the 
size of the submatrix in (fl"8l) , leading to higher complexity for the ACG-ALP decoding 
algorithm. 

To avoid such numerical problems in our implementation of the CSA, we used 1 — 10 6 
instead of 1 on the right-hand side of the inequality in line 14 of Algorithm \T\ Whenever the 
LP solver outputs a solution vector, coordinates with value less than 10~ 6 were rounded to 
and coordinates with value larger than 1 — 1CT 6 were rounded to 1. The rounded values were 
then used in the cut-search and RPC-generation steps in the decoding algorithms described in 
previous sections. If such a procedure were not applied, and if, as a result, false cuts were to be 
produced, the corresponding constraints, when added into the LP problem to be solved in the 
next step, would leave the solution vector unchanged, causing the decoder to become stuck in an 
endless loop. We saw no such behavior in our decoder simulations incorporating the prescribed 
thresholding operations. 

Finally, we want to point out that there exist LP solvers, such as QSopt_ex Rational LP Solver 
(221, that produce exact rational solutions to LP instances with rational input. However, such 
solvers generally have higher computational overhead than their floating-point counterparts. For 
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E b /N (dB) 



Fig. 1. FER versus E b /N for random (3,4)-regular LDPC code of length 100 on the AWGN channel. 



this reason, we did not use an exact rational LP solver in our empirical studies. 

Fig. [T] shows the simulation results for the length-100, regular-(3,4) LDPC code whose FER 
performance was also evalutated in [[T2l and |fT3l . We can see that the proposed algorithms have a 
gain of about 2 dB over the original LP and ALP decoder. They also perform significantly better 
than both the separation algorithm and the random-walk algorithm. The figure also shows the 
results obtained with the Box-and-Match soft-decision decoding algorithm (BMA) 11231 , whose 
FER performance is guaranteed to be within a factor of 1.05 times that of ML decoding. We 
conclude that the performance gap between the proposed decoders and ML decoding is less than 
0.2 dB at an FER of KT 5 . 

In Fig. [2l we show simulation results for MacKay's length-96, (3,6)-regular LDPC code (the 
96.33.964 code from Il20"l ). Again, the proposed ALP-based decoders with ACG demonstrate 
superior performance to the original LP, BP, and SA decoders over the range of SNRs considered. 
Table [I] shows the actual frame error counts for the ACG-ALP decoder, with frame errors 
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Fig. 2. FER versus E^/Nq for MacKay's random (3,6)-regular LDPC code of length 96 on the AWGN channel. 

TABLE I 

Frame Errors of ACG-ALP decoder on MacKay's random (3,6)-regular LDPC code of length 96 on the 

AWGN CHANNEL. 



E b /N (dB) 


Transmitted Frames 


Error Frames 


Pseudocodewords 


Incorrect Codewords 


3.0 


1,136,597 


3,000 


857 


2,143 


3.5 


4,569,667 


3,000 


395 


2,605 


4.0 


16,724,921 


3,000 


103 


2,897 


4.5 


54,952,664 


3,000 


12 


2,988 


5.0 


185,366,246 


3,000 





3,000 


5.5 


665,851,530 


3,000 





3,000 



classified as either pseudocodewords or incorrect codewords; the ACG-MALP-B and ACG- 
MALP-C decoder simulations yielded very similar results. We used these counts to obtain a 
lower bound on ML decoder performance, also shown in the figure, by dividing the number of 
times the ACG-ALP decoder converged to an incorrect codeword by the total number of frames 
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Fig. 3. FER versus E b /N for (155,64) Tanner LDPC code on the AWGN channel. 



transmitted. Since the ML certificate property of LP decoding implies that ML decoding would 
have produced the same incorrect codeword in all of these instances, this ratio represents a lower 
bound on the FER of the ML decoder. We note that, when E b /N is greater than 4.5 dB, all 
decoding errors correspond to incorrect codewords, indicating that the ACG-ALP decoder has 
achieved ML decoding performance for the transmitted frames. 

Fig. [3] compares the performance of several different decoders applied to the (3,5)-regular, 
(155,64) Tanner code, as well as the ML performance curve from [fT3ll . It can be seen that the 
proposed ACG-ALP-based algorithms narrow the 1.25 dB gap between the original LP decoding 
and ML decoding to approximately 0.25 dB. 

We also considered two longer codes, MacKay's rate-|, random (3,6)-regular LDPC code 
of length 408 (the 408.33.844 code from flU) and a rate-0.89 LDPC code of length 999 (the 
999.111.3.5543 code from [20]). Because of the increased complexity of the constituent LP 
problems, we only simulated the ACG-MALP-B and ACG-MALP-C decoders. In Fig. H it is 
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Fig. 4. FER versus Eb/No for MacKay's random (3,6)-regular LDPC code of length 408 on the AWGN channel. 




Fig. 5. FER versus Eb/No for MacKay's random (3,27)-regular LDPC code of length 999 on the AWGN channel. 
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confirmed that the proposed decoding algorithms provide significant gain over the original LP 
decoder and the BP decoder, especially in the high-SNR region. The results for the high-rate 
LDPC code, as shown in Fig. [51 again show that the proposed decoding algorithms approaches 
ML decoding performance for some codes, where the ML lower bound is obtained using the 
same technique as in Fig. [2l However, for the code of length 408, we found that the majority of 
decoding failures corresponded to pseudocodewords, so, in constrast to the case of the length-96 
and length-999 MacKay codes discussed above, the frame error data do not provide a good 
lower bound on ML decoder performance to use as a benchmark. 

Since the observed improvements in ACG-ALP-based decoder performance comes from the 
additional RPC cuts found in each iteration, these decoding algorithms generally require more 
iterations and/or the solution of larger LP problems in comparison to ALP decoding. In the 
remainder of this section, we empirically investigate the relative complexity of our proposed 
algorithms in terms of such statistics as the average number of iterations, the average size of 
constituent LP problems, and the average number of cuts found in each iteration. All statistical 
data presented here were obtained from simulations of the Tanner (155,64) code on the AWGN 
channel. We ran all simulations until at least 200 frame errors were counted. 

In Fig. [6l we compare the average number of iterations needed, i.e., the average number 



of LP problems solved, to decode one codeword. Fig. |7(a)| compares the average number of 
constraints in the LP problem of the final iteration that results in either a valid codeword or a 
pseudocodeword with no more cuts to be found. In Fig. |7(b)[ we show the average number of 



cuts found and added into the LP problem in each iteration. Fig. |7(c)| and Fig. |7(d)| show the 



average number of cuts found from the original parity-check matrix H and from the generated 
RPCs, respectively. 

From Fig. |6] and Fig. |7(a)| we can see that, as expected, the ACG-ALP decoder takes fewer 



iterations to decode a codeword on average than the ACG-MALP-B/C decoders, but the ACG- 
MALP-B/C decoders have fewer constraints in each iteration, including the final iteration. We 
have observed that the ACG-MALP-B/C decoders require a larger number of iterations to decode 
than the ACG-ALP decoder, and fewer cuts are added into the constituent LP problems in each 
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Fig. 6. Average number of iterations for decoding one codeword of (155,64) Tanner LDPC code. 

TABLE II 

The average accumulated number of constraints in all iterations of decoding one codeword of (155,64) 

Tanner code on the AWGN channel 



E b /N (dB) 


ACG-ALP 


ACG-MALP-B 


ACG-MALP-C 


1.83 


5495.8 


5223.3 


4643.1 


2.33 


1401.2 


1387.3 


1217.0 


2.83 


339.7 


326.9 


300.9 


3.33 


111.0 


106.4 


105.4 


3.83 


64.3 


58.8 


62.8 



iteration on average, as reflected in Fig. |7(b)| This is because there are some iterations in 
which the added constraints had been previously removed. Among all three proposed ACG- 
based decoding algorithms, we can see that the ACG-ALP decoder has the largest number of 
constraints in the final iteration and needs the least overall number of iterations to decode, while 
ACG-MALP-B decoding has the smallest number of constraints but requires the largest number 




Fig. 7. Average number of constraints/cuts during decoding iterations for decoding one frame of (155,64) Tanner LDPC code. 



of iterations. The ACG-MALP-C decoder offers a tradeoff between those two: it has fewer 
constraints than the ACG-ALP decoder and requires fewer iterations than the ACG-MALP-B 
decoder. If we use the accumulated number of constraints in all iterations to decode one codeword 
as a criterion to judge the efficiency of these algorithms during simulation, then ACG-MALP-C 
decoding is more efficient than the other two algorithms in the low and moderate SNR regions, 
as shown in Table Hfl Note that the ACG-MALP-B decoder is most efficient at high SNR where 
the decoding of most codewords succeeds in a few iterations and the chance of a previously 
removed inactive constraint being added back in later iterations is quite small. Hence, ACG- 
MALP-B decoding is preferred in the high-SNR region. 

Fig. [8] presents an alternative way of comparing the complexity of the decoding algorithms. 
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Fig. 8. Average simulation time for decoding one codeword of (155,64) Tanner LDPC code. 



It shows the average decoding time when we implement the algorithms using C++ code on a 
desktop PC, with GLPK as the LP solver. The BP decoder is implemented in software with 
messages represented as double-precision floating-point numbers, and the exact computation of 
sum-product algorithm is used, without any simplification or approximation. The BP decoder 
iterations stop as soon as a codeword is found, or when the maximum allowable number of 
iterations - here set to 100 and 1000 - have been attempted without convergence. The simulation 
time is averaged over the number of transmitted codewords required for the decoder to fail on 
200 codewords. 

We observe that the ACG-MALP-B and ACG-MALP-C decoders are both uniformly faster 
than ACG-ALP over the range of SNR values considered, and, as expected from Table HH 
ACG-MALP-C decoding is slightly more efficient than ACG-MALP-B decoding in terms of 
actual running time. Of course, the decoding time depends both on the number of LP problems 
solved and the size of these LP problems, and the preferred trade-off depends heavily upon the 
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implementation, particularly the LP solver that is used. Obviously, the improvement in error-rate 
performance provided by all three ACG-based decoding algorithms over the ALP decoding comes 
at the cost of increased decoding complexity. As SNR increases, however, the average decoding 
complexity per codeword of the proposed algorithms approaches that of the ALP decoder. This is 
because, at higher SNR, the decoders can often successfully decode the received frames without 
generating RPC cuts. 

Fig. [6] shows that the ACG-ALP decoder requires, on average, more iterations than the SA 
decoder. Our observations suggest that this is a result of the fact that the ACG-ALP decoder 
can continue to generate new RPC cuts after the number of iterations at which the SA decoder 
can no longer do so and, hence, stops decoding. The simulation data showed that the additional 
iterations of the ACG-ALP decoder often resulted in a valid codeword, thus contributing to its 
superiority in perfomance relative to the SA decoder. 

From Fig. |7(b)| it can be seen that the ACG-ALP-based decoding algorithms generate, on 



average, fewer cuts per iteration than the SA decoder. Moreover, as reflected in Fig. |7(c)| and 
|7(d)[ the ACG-ALP decoders find more cuts from the original parity-check matrix and generate 
fewer RPC cuts per codeword. These observations suggest that the CSA is very efficient in 
finding cuts from a given parity check, while the SA decoder tends to generate RPCs even when 
there are still some cuts other than the Gomory cuts that can be found from the original parity- 
check matrix. This accounts for the fact, reflected in Fig. [8] that the SA becomes less efficient 
as SNR increases, when the original parity-check matrix usually can provide enough cuts to 
decode a codeword. The effectiveness of our cut-search algorithm permits the ACG-ALP-based 
decoders to successfully decode most codewords in the high- SNR region without generating 
RPCs, resulting in better overall decoder efficiency. 

Due to limitations on our computing capability, we have not yet tested our proposed algorithms 
on LDPC codes of length greater than 1000. We note that, in contrast to lfT2l and [[T6l , we cannot 
give an upper bound on the maximum number of iterations required by the ACG-ALP-based 
decoding algorithms because RPCs and their corresponding parity inequalities are generated 
adaptively as a function of intermediate pseudocodewords arising during the decoding process. 
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Consequently, even though the decoding of short-to-moderate length LDPC codes was found 
empirically to converge after an acceptable number of inter ations, some sort of constraint on the 
maximum number of iterations allowed may have to be imposed when decoding longer codes. 
Finally, we point out that the complexity of the algorithm for generating cut-inducing RPCs lies 
mainly in the Gaussian elimination step, but as applied to binary matrices, this requires only 
logical operations which can be executed quite efficiently. 

VI. Conclusion 

In this paper, we derived a new necessary condition and a new sufficient condition for a 
parity-check constraint in a linear block code parity-check matrix to provide a violated parity 
inequality, or cut, at a pseudocodeword produced by LP decoding. Using these results, we 
presented an efficient algorithm to search for such cuts and proposed an effective approach to 
generating cut-inducing redundant parity checks (RPCs). The key innovation in the cut-generating 
approach is a particular transformation of the parity-check matrix used in the definition of the 
LP decoding problem. By properly re-ordering the columns of the original parity-check matrix 
and transforming the resulting matrix into a "partial" reduced row echelon form, we could 
efficiently identify RPC cuts that were found empirically to significantly improve the LP decoder 
performance. We combined the new cut-generation technique with three variations of adaptive 
LP decoding, providing a tradeoff between the number of iterations required and the number of 
constraints in the constituent LP problems. Frame-error-rate (FER) simulation results for several 
LDPC codes of length up to 999 show that the proposed adaptive cut-generation, adaptive LP 
(ACG-ALP) decoding algorithms outperform other enhanced LP decoders, such as the separation 
algorithm (SA) decoder, and significantly narrow the gap to ML decoding performance for LDPC 
codes with short-to-moderate block lengths. 
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